Picture and sound decoding apparatus picture and sound encoding apparatus and information transmission system

ABSTRACT

In order to control the coding quantity depending on the processing situation at the terminal when decoding or synthesizing plural pictures or sounds simultaneously, the decoding apparatus of the invention comprises a reception control unit  11  for receiving the information, a separation unit  12  for analyzing and separating the received information, a priority decision unit  14  for determining the priority of processing of the pictures separated in the separation unit  12 , a picture expanding unit  18  for expanding the pictures according to the determined priority, s picture synthesizing unit  19  for synthesizing the pictures on the basis of the expanded pictures, a synthesis result accumulating unit  22  for accumulating the synthesized pictures, a reproduction time control unit  23  for controlling the time for starting reproduction, and an output unit  24  for delivering the result of synthesis according to the information of the reproduction time control unit  23.

This application is a U.S. National Phase Application of PCTInternational Application PCT/JP97/02696.

TECHNICAL FIELD

The present invention relates to picture and sound decoding apparatus,picture and sound coding apparatus, and information transmission systemfor decoding, coding and synthesizing plural pictures and soundssimultaneously.

BACKGROUND ART

Hitherto there was a video communication system with a feel of presenceby satisfying the sensation of the existence of a person in front ofoneself, by extracting, for example, a human figure from the image ofthe space scene of one's own side, and superposing and displaying thisimage, and a human image sent from the opposite side, and a virtualspace image preliminarily stored for displaying commonly with theopposite side (Japanese Patent Publication No. 4-24914, Hypermediasystem personal communication system; Fukuda, K., Tahara, T., Miyoshi,T.: “Hypermedia Personal Computer Communication System: FujitsuHabitat,” FUJITSU Sci. Tech. J., 26, 3, pp. 196–206, October 1990;Nakamura: “Distributed cooperative job support by virtual realitysensation through network,” Audio-Visual Compound Information ProcessingResearch Group, Information Processing Society of Japan, 1993). In theprior art, in particular, in order to synthesize the images, methodsabout increasing the speed and reducing the memory capacity have beeninvented (for example, Japanese Patent Publication No. 5-46592, Imagesynthesizing apparatus; Japanese Laid-open Patent No. 6-105226, Imagesynthesizing apparatus).

In the prior art, however, image synthesizing systems for synthesizingtwo-dimensional still pictures and three-dimensional CG data have beenproposed, but nothing has been mentioned about a method of realizing asystem for synthesizing and displaying by decoding (expanding) pluralmotion pictures or sounds simultaneously. In particular, in the terminaldevice for simultaneously decoding, synthesizing and displaying pluralpictures and sounds, nothing has been mentioned about the reproducingmethod of picture and sound not leading to breakdown due to shortage ofterminal capacity or fluctuation of processing capacity. In addition,nothing has been discussed about the method of decoding, synthesizingand displaying plural pictures depending on the accounting situation.

More specifically, nothing has been considered about:

(1) method of controlling plural pictures, sound information,information describing the relation of plural pictures and sounds, andinformation of processing result; and

(2) method about determining method of priority of decoding,synthesizing and displaying of plural pictures and sounds when theterminal processing state is overloaded, and also reproducing andaccounting.

Further, in the environments of decoding, synthesizing and displayingplural pictures and sounds simultaneously, nothing has been consideredabout the method of controlling the quantity of coding by varying themethod of compression of image depending on the state at the receivingterminal side or the priority of decoding, synthesizing, and displayingat the receiving terminal.

DISCLOSURE OF THE INVENTION

Considering these problems of the prior art, it is an object of thepresent invention to present a picture and sound decoding apparatus, apicture and sound coding apparatus, and an information transmissionsystem capable of controlling the quantity of coding depending on theprocessing situation at the terminal when decoding or synthesizingplural pictures and sounds simultaneously, and capable of controllingdecoding, synthesizing and displaying of plural pictures and soundsdepending on the accounting situation.

The invention is not limited to synthesis of two-dimensional imagesalone. It may include an expression format combining two-dimensionalimage and three-dimensional image, and an image synthesizing method forsynthesizing an image by combining plural adjacent images such aspanoramic images.

The communication format of the invention is not limited to the wiredtwo-way CATV or B-ISDN alone. For example, it may include radio wavetransmission (for example, VHF band, UHF band) or satellite broadcastingfor picture and sound from the center side terminal to the householdside terminal, and analog telephone circuit or N-ISDN for informationtransmission from the household side terminal to the center sideterminal (picture, sound and data are not always required to bemultiplexed). It may further include the communication format making useof wireless means, such as IrDA, PHS (personal handy phone), and radioLAN.

Applicable terminals include portable terminals such as portableinformation terminals, and desk-top terminals such as set-top box andpersonal computer.

One aspect of the present invention is a picture decoding and codingapparatus comprising a picture coding apparatus including picture codingmeans for coding picture information, and transmission control means fortransmitting or recording the coded various information, and a picturedecoding apparatus including reception control means for receiving thecoded various information, picture decoding means for decoding receivedvarious information, picture synthesizing means for synthesizing one ormore decoded pictures, and output means for delivering the synthesizedpicture.

Another aspect of the present invention is a sound decoding and codingapparatus comprising a sound coding apparatus including sound codingmeans for coding sound information, and transmission control means fortransmitting or recording the coded various information, and a sounddecoding apparatus including reception control means for receiving thecoded various information, sound decoding means for decoding receivedvarious information, sound synthesizing means for synthesizing one ormore decoded sounds, and output means for delivering the synthesizedsound.

Still another aspect of the present invention is a realtime picturecoding apparatus comprising one or more picture input means for feedingpictures, picture input control means for controlling the control stateof the picture input means, other terminal control request control meansfor controlling the reception state of a reception terminal, codingprocess decision means for determining the coding method of picturesdepending on at least the controlled reception state of the receptionterminal or the control state of said picture input means, picturecoding means for coding said input picture according to the result ofdecision by the coding process decision means, and output means fordelivering the coded picture.

A further aspect of the present invention is an information transmissionsystem, being an information transmission system using at least one ofthe picture decoding and coding apparatus and the sound decoding andcoding apparatus as the reception terminal, and at least one of thepicture decoding and coding apparatus the sound decoding and codingapparatus and the real time picture coding apparatus as the transmissionterminal, and connecting these terminals through a communication route,wherein one of the presence or absence of coding of pictures or sounds,priority of coding, coding method, picture size to be coded, value ofquantization step, number of frames, and priority of processing when thereception terminal is overloaded is determined at the transmissionterminal, by transmitting at least one of the load of the receptionterminal, the information about the priority of coded information to beprocessed as being determined in the priority decision means at thereception terminal, and the frame skip situation at the receptionterminal, to the transmission terminal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic structural diagram of a picture decoding andcoding apparatus in an embodiment of the invention;

FIG. 2 is a schematic structural diagram of a picture and sound decodingand coding apparatus showing other example in the same embodiment;

FIG. 3 is a diagram for explaining an example of adding informationabout priority in a communication and recording format;

FIG. 4 is a diagram for explaining an example of composing the inventionby software;

FIG. 5 is a diagram for explaining the composition of information;

FIG. 6 is a diagram for explaining the operation of DEMUX thread.

FIG. 7 is a diagram for explaining the operation of monitor thread.

FIG. 8 is a diagram for explaining the operation of decoding process;

FIG. 9 is a diagram for explaining the operation of picture synthesizingthread;

FIG. 10 is a diagram for explaining the operation of display monitorthread;

FIG. 11 is a diagram for explaining the user interface picturesynthesizing apparatus;

FIG. 12 is a diagram for explaining the method of picture transmissiondepending on fluctuation of capacity of the receiving side terminal;

FIG. 13 is a diagram for explaining the picture compression apparatus inan embodiment of the invention;

FIG. 14 is a diagram for explaining the information controlled byoperation control unit;

FIG. 15 is a diagram for explaining the picture compression apparatusfor creating panoramic pictures; and

FIG. 16 is a diagram for explaining the response situation betweentransmission terminal and reception terminal.

REFERENCE NUMERALS

-   11 Reception control unit-   12 Separation unit-   13 Transmission control unit-   14 Priority decision unit-   17 Time information control unit-   18 Picture expanding unit-   19 Picture synthesizing unit-   20 Sound expanding unit-   21 Sound synthesizing unit-   31 DEMUX thread-   36 Decoding process-   37 Monitor thread-   39 Picture synthesizing thread-   42 Display monitor thread-   1204 Operation control unit-   1205 Picture compression unit-   1208 Picture processing decision control means-   1401 Priority decision control unit-   1402 Operation history control unit-   1404 Picture synthesizing unit-   1407 Input unit

MODE FOR CARRYING OUT THE INVENTION

The invention is described below by referring to drawings showing itspreferred embodiments. The term “picture” used in the invention includesboth still picture and motion picture. The applicable pictures may be amixture of two-dimensional picture such as computer graphics (CG) andthree-dimensional picture as composed from a wire-frame model. In such acase, the relation between pictures corresponds to the wire-frame model.The script language for describing may include JAVA, VRML, etc, whichlanguage is incorporated herein by reference.

FIG. 1 and FIG. 2 are schematic structural diagrams of picture decodingand coding apparatus in an embodiment of the invention. FIG. 1 shows aconstitution without sound reproducing function, and FIG. 2 shows aconstitution having both picture and sound reproducing function. Ofcourse, it may be constituted similarly for sound alone.

This apparatus shown in FIG. 1 or FIG. 2 is composed of a codingapparatus and a decoding apparatus, and the coding apparatus in FIG. 1comprises a priority providing unit 101 for determining the priority ofprocessing of coded picture at the time of overloading, andcorresponding between the coded image and the priority, a picture codingunit 102 for coding the picture, a transmission control unit 103 fortransmitting or recording coded information provided with priority, anda reception control unit 104 for receiving coded information. The codingapparatus in FIG. 2 further comprises a sound coding unit 105 for codingthe sound.

On the other hand, in the decoding apparatus, a reception control unit11 for receiving information and a transmission control unit 13 fortransmitting information are means for transmitting information incoaxial cable, CATV, LAN, modem, etc. Connection formats of terminalinclude two-way sending and receiving format of video informationbetween terminals such as TV telephone and TV conference system, andbroadcasting type (one-way) picture broadcasting format by satellitebroadcasting, CATV or Internet. In the invention, such connectionformats of terminals are taken into consideration.

A separation unit 12 is means for analyzing and separating the coded(compressed) reception signal (in the case of compression apparatus, itis a reverse operation, and hence it corresponds to a multiplexingunit). For example, in MPEG1, MPEG2, and H. 320 terminal (regulation ofTV telephone and conference apparatus using N-ISDN), the H. 221 is theregulation for multiplexing and separating the video/audio/data, and inthe H. 324 terminal (regulation of TV telephone and conference apparatususing analog telephone circuit), the H. 223 is the correspondingregulation MPEG 1 and MPEG 2 Standards and the InternationalTelecommunications Union (ITU-T) H.-series standards are allincorporated herein by reference. The invention may be realized eitherby the constitution conforming to the regulation or by the constitutionnot conforming to the regulation. Or, as done in the H. 323 or Internet,the picture and sound may be transmitted independently by differentstreams.

A priority decision unit 14 determines the priority of decoding(hereinafter referred to as “expanding”) the information obtained fromthe separation unit 12 (for example, picture, sound, controlinformation) when the terminal is overloaded in the following method,and expanding the picture or expanding the sound (in the method ofdetermining the priority of processing, it may be preliminarily decidedin the reception terminal device, or the information about the prioritydetermined in the following method may be added to the recording mediaor transmission packet at the transmission side terminal (codingapparatus), and transmitted and added as recording format; as the methodof expressing the priority, the priority may be expressed bynon-numerical method such as large, medium and small, or by numericalmethod such as 1, 2, 3).

Using the identifier for handling data in the stream unit composed ofplural picture or sound frames, by processing transmission and receptionof data between the transmission side and reception side, control ofbuffer at the reception side, and scheduling of transmission of data attransmission side may be realized. That is, by noticing the identifierof the stream sent from the transmission side as required, the receivingsituation at the reception side may be investigated, or the identifierof the stream not requiring may be noticed to the reception terminal, orthe required stream may be demanded from the reception side.

The priority providing means for determining the priority of processingof coded information at the time of overloading by the standardmentioned above, and corresponding between the coded information and thedetermined priority may be provided in the picture coding apparatus orsound coding apparatus, and according to the priority of variousreceiving information at the time of overloading, the picture frame orsound of the priority to be processed may be determined by the prioritydecision means for determining the method of processing, so that codingor synthesizing process may be done. As for the picture frame, it isnecessary to insert periodically the frame undergoing intraframe coding(I frame) so that the frame can be skipped.

The unit for providing the priority may be each unit of picture or sound(comparison of priority between frames), or the stream unit composed ofplural frames (comparison of priority between streams).

The method of taking note of the feature of image includes the methodson the basis of the image compression format (for example, between H.263 and run length, the priority is given to the run length), picturesize (for example, between CIF and QCIF, the priority is given to theQCIF), contrast (for example, the priority is given to the highercontrast), picture synthesizing ratio (for example, the priority isgiven to the higher synthesizing ratio), quantization step (for example,the priority is given to the smaller quantization step), differencebetween inter-frame coding and intra-frame coding (for example, thepriority is given to the intra-frame coding), display position (forexample, the priority is given to the middle display position; or in thecase of three-dimensional picture, the priority is low when the pictureis set at inner side, and the priority is set high when displayed at thefront side), frame number (the priority is high in first frame and finalframe, or the priority is heightened for the frame of scene change,etc.), the number of frames (for example, the priority is higher in thepicture smaller in the number of frames to be reproduced; the framenumber corresponds to temporary reference (TR) in the case of H. 263,and it may be judged according to the change of TR value), the voicedsection and silent section, the display time (PTS), and decoding time(DTS).

In addition, the same priority is given to the P frame and B frame byinter-frame coding. In the intra-frame coded pictures, by assigningpriority in plural stages, the skipping frequency can be controlled.

Examples taking note of difference in media include the method ofexpanding the sound by priority over expanding of picture. As a result,the sound can be reproduced without interruption.

Furthermore, on the basis of the reproduction permit informationcontrolled at the reception side terminal, the information to beexpanded (picture, sound) may be determined, or the information to beexpanded may be selected on the basis of the reproduction permitinformation sent from the transmission side as control information. Thereproduction permit information specifically includes the accountinginformation (for example, if not paid duly, processing of expanding,synthesizing or display is not done; the accounting information can becontrolled at the reception terminal side, or the accounting informationmay be controlled at the transmission side), the service contentinformation (for example, in the case of an adult broadcast program,unless reproduction is permitted at the terminal side, processing ofexpanding, synthesizing or display is not done; the reproduction permitmay be controlled at the reception side terminal or at the transmissionside terminal), the password (for example, a specific program is notexpanded, synthesized or displayed unless the password is entered; thepassword may be controlled at the reception side terminal or at thetransmission side terminal), the user code (for example, unless the useris permitted, processing of expanding, synthesizing or display is notdone; the user code may be controlled at the reception side terminal orat the transmission side terminal), and the nation code (for example,the picture and sound to be expanded, synthesized and displayed, and thereproduction method are changed depending on the nation; the nation codemay be controlled at the transmission side or at the reception side;scrambling is realized by changing the reproduction method by the nationcode).

The reproduction method by limiting the permission of reproduction ofpicture and sound by the accounting information, service contentinformation, password, or user code includes the method of deviating theposition or pixel intentionally when synthesizing and displaying thepicture, changing the magnification or contraction of picture orsampling of picture (for example, by low pass), inverting the pixels,changing the contrast, changing the color pallets, and skipping theframes. In these picture reproducing methods (expanding, synthesizingand display pictures), each frame may be restricted. Or, in the unit ofGOB (group of block) which is an independent processing unit smallerthan one frame, as defined by H. 263 which is one of picture compressiontechniques, the picture expanding, synthesizing and display method maybe restricted, so that more flexible control is realized as comparedwith the conventional technique of disturbing the entire picture. Thatis, by processing in the GOB unit, only a part of the picture can bescrambled, and the interactive software such as the software using theimage synthesis can be evaluated.

Similarly, sound reproducing methods include the methods of changing thesound level, changing the sound direction, changing the sound frequency,changing the sound sampling, and inserting different picture or sound(in any method, it is divided into the method of processingpreliminarily at the transmission side, and the method of processing atthe reception side).

The reproducing method of picture and sound includes a method ofcanceling the synchronism of picture and sound. The priority of pictureand sound to be expanded, or the presence or absence may be determinedby the information indicating the rank of synthesizing and displaying(the display sequence is preliminarily determined at the reception sideterminal, for example, the priority is given to the CIF or stillpicture, or the display sequence is added preliminarily to thetransmission information at the transmission side as the informationabout priority), the information indicating the rank of expanding (theexpanding sequence is preliminarily determined at the reception sideterminal, for example, the priority is given to the QCIF or intra-framecoded picture data, or the speech sound is expanded by priority overBGM; similarly, the display sequence is added to the transmissioninformation at the transmission side), the user's instruction (forexample, the picture or sound information to be expanded, synthesized ordisplayed is selected by the user's instruction, or the picture or soundinformation to be expanded, synthesized or displayed is determined onthe basis the information selected according to the request), theterminal processing capacity (for example, by measuring the occupationtime of CPU processing in a specific period at the present or in thepast, expanding, synthesizing or displaying of the picture or soundsupposed to take time is restricted; as the estimating method ofprocessing time, the time required for local decoding when compressing,or the time required for compression is controlled in relation to thecompressed picture information, so that the presence or absence ofexpanding, synthesizing or displaying, or the priority can bedetermined), the reproduction time (for example, expanding, synthesizingor displaying of the picture or sound information passing thereproduction time is stopped), or the decoding time.

In addition, as the method for preventing specific picture or sound frombeing expanded or displayed by priority, on the basis of the informationrelating to the execution rate for processing of expanding, synthesizingor displaying of picture or sound, the sequence or presence or absenceof the picture to be expanded, synthesized or displayed can bedetermined. For example, once out of ten times of expanding, it is setat the reception terminal side that the picture of CIF size is expanded,or by specifying the execution rate of expanding, synthesizing ordisplaying the picture or sound at the transmission side, the pictureinformation or sound information can be transmitted according to this.The execution rate is specifically defined by the insertion interval ofI frame (intra-frame coded frame). As a result, expanding, synthesizingor displaying of specific picture or sound object only is prevented.

Such information about the priority for controlling expanding,synthesizing or displaying may be added not only at the transmissionside apparatus, but may be also added and controlled at a relayingapparatus. Moreover, by transmitting the information about the prioritydetermined in the priority decision unit 14 of the decoding apparatus atthe reception terminal, to the transmission destination through thetransmission control unit 13, the picture or sound can be transmitteddepending on the situation of determination in the priority decisionunit 14 (by sending the ID of the picture object that is hardly selectedto the transmission side, wasteful transmission is prevented).Incidentally, the information showing the priority of processing whenthe reception terminal is overloaded may be determined at the receptionterminal apparatus, or may be transmitted as transmission format, or thetransport stream of MPEG2 may be expanded as the format for recording inthe recording media such as CD-ROM or hard disk, or the transmission andrecording format without consideration of standardization may be alsoemployed. Or, using different streams in individual media (picture,sound, information describing the relation of picture and sound), datamay be transmitted and recorded without multiplexing.

The picture expanding unit 18 as picture decoding means is means forexpanding the picture (hereinafter coding means in the case of thecoding apparatus), and the picture formats handled in the pictureexpanding unit 18 include MPEG1, MPEG2, H. 261, H. 263, and others. Thepicture may be expanded in the unit of one frame, or it may be processedin the unit of GOB specified in H. 263. When processing in the unit ofone frame, in the case of inter-frame coding, the expanded state of theprevious frame must be stored in the picture expanding unit 18. When thepicture is expanded in the GOB unit, the sequence relation of expandingthe pictures does not matter. Therefore, when expanding in the GOB unit,plural picture expanding units 18 are not required in the receivingapparatus, and plural pictures can be expanded by one picture expandingunit 18. To the contrary, the results of expanding must be accumulated.

The sound expanding unit 20 as the sound decoding means in FIG. 2 is themeans for expanding the sound, and the sound format handled in the soundexpanding unit 20 includes G. 721 and G. 723. The method for processingincludes the software processing by DSP or general-purpose CPU, andprocessing by special-purpose hardware.

When realizing by the software, expanding process of picture and soundis individually controlled in the unit of one process or thread, andwhen there are simultaneously plural pictures or sounds to be expanded,they are processed by dividing the time by the number of processes orthreads in the range to be processed.

The picture expanding control unit 15 is means for controlling theexpanded state of picture. The sound expanding control unit 16 is meansfor controlling the expanded state of sound. For example, when realizingthese control units by the software, the compressed information obtainedfrom the separation unit 12 is transferred to the picture expanding unit18 and sound expanding unit 20 in the predetermined sequence (forexample, executing first in the sound expanding unit 20 and thenexecuting in the picture expanding unit 18), and the expanded state iscontrolled. When all expanding is over, the expanded information istransferred to the picture synthesizing unit 19 or sound synthesizingunit 21. In the software, by using shared memory and semaphore, thetransferred information is limited, or end of expanding process is known(detail is described later).

A time information control unit 17 is means for controlling theinformation about the time. For example, when realizing the system by apersonal computer, the time information can be realized by utilizing thetimer of the personal computer.

The picture synthesizing unit 19 synthesizes a picture on the basis ofthe expanded image data. When synthesizing plural pictures, the picturesare synthesized on the basis of the synthesis ratio (α value) of eachpicture. For example, when synthesizing two pictures, if the synthesisratio of the foreground picture is α, the RGB value of the backgroundpicture is mixed at a rate of 1−α, and the foreground picture, at α.Incidentally, the picture to be expanded is controlled in the processingin the unit of one frame, and therefore when synthesizing pluralpictures by using the display time, the system configuration andmounting can be simplified. Until discarding of the expanding result isinstructed from the transmission side in the picture synthesizing unit19 or sound synthesizing unit 21, by holding, controlling and utilizingthe expanding result, it is not necessary to send the information ofsame pattern repeatedly from the transmission side.

On the basis of the information describing the relation of mutualpictures or mutual sounds, when combining the picture or sound, bypresenting that the pictures or sounds that cannot be synthesized arepresent as necessary decoded pictures or sounds are not ready, the usercan know the state of synthesis. Accordingly, the user instructs toselect the necessary picture quality or select the picture desired to besynthesized beforehand, so that necessary information can be synthesizedwithout dropout. Incidentally, the method of accumulating andcontrolling the decoded picture and sound data in the buffer includesthe methods of erasing from the older data in the sequence of arrival,and erasing by observing the script describing the relation of mutualpictures or mutual sounds, and also observing the state of use of thedecoded picture or sound data on the whole.

The sound expanding control unit 16 controls the expanding state of atleast one sound expanding unit 20 for expanding the sound.

The sound synthesizing unit 21 is means for synthesizing the sound onthe basis of the expanded information, and a synthesized resultaccumulating unit 22 is means for accumulating the picture synthesizedby the picture synthesizing unit 19 and the sound synthesized by thesound synthesizing unit 21.

A reproduction time control unit 23 is means for reproducing thesynthesized picture or sound at the time for starting reproduction.

An output unit 24 is means for issuing the synthesized result (forexample, display and printer), and an input unit 25 is means forentering information (for example, keyboard, mouse, camera, video). Aterminal control unit 26 is means for controlling these units.

FIG. 3 is a diagram for explaining an example of a case of addinginformation about priority in communication and recording format.

FIG. 3( a) shows an example of multiplexing all media completely(picture, sound, control information). The control information includesthe priority for determining the processing when overloaded (thepriority as indicated in the invention), and the priority showing thesequence of display. The control information may also include theinformation about the relation between pictures, between sounds, orbetween picture and sound (in terms of time or position). The example inFIG. 3 (a) is suited, for example, to application in multiplexing ofMPEG1/2, or packet multiplexing mixing control information and data(picture, sound) such as H. 223. The priority of processing in overloadis added in the frame unit or stream unit.

FIG. 3( b) shows an example of multiplexing information in every media.In this example, the control information, picture information, and soundinformation are transmitted from individual communication ports. Theinformation about the relation between pictures, between sounds, andbetween picture and sound may be transmitted as control information fromdifferent communication port from that of picture or sound. It is suitedto application where plural communication ports can be establishedsimultaneously, such as in H. 323 and Internet, and as compared withFIG. 3( a), multiplexing process can be simplified, so that the terminalload can be alleviated.

As the method for describing mutual pictures or mutual sounds, it may beconsidered to be applicable by descriptive language such as JAVA andVRML, but the specification of descriptive language of the script maynot be determined automatically. Accordingly, by adding an identifierfor identifying the descriptive method of the information describing therelation between pictures or between sounds (for example, information interms of position or time, such as display period), it is applicable toplural methods of description. To add the identifier for identifying theinformation descriptive method, for example, in MPEG2, it is provided inthe program map table for controlling the stream of MPEG2-TS, or in thestream describing the script. The priority of processing in overload isadded together with the information describing the correspondingrelation of picture and sound (control information). In the MPEG2, inorder to control by the program map table for relating the video streamand audio stream of MPEG2-TS (transport stream), the structuralinformation and stream for relating between picture and sound may bedefined and controlled, so that it can be transmitted independently ofthe data also in the MPEG2.

FIG. 4 is a diagram explaining an example of composing the invention bythe software. When the invention is realized on the operating systemcapable of operating in multitask, each process explained in FIG. 1 andFIG. 2 is divided into execution module units of software such asprocess and thread, and between processes or between threads,information is exchanged by shared memory, and the shared information iscontrolled exclusively by the semaphore (in the example in FIG. 4, thearea indicated by solid line corresponds to the semaphore). The functionof each process and thread is described below.

A DEMUX thread 31 reads the information (picture, sound, controlinformation) multiplexed from the network or disk, and separates into amonitor table (detail given below) describing the corresponding relationof sounds, pictures, and sound and picture and the information aboutreproduction time. The DEMUX thread 31 corresponds to the separationunit 12. The information separated by the DEMUX thread 31 is sent outinto a sound ring buffer 32, a picture ring buffer 33, and a monitorring buffer 34. In the case of sound information, the information sentout into the ring buffer 32 is expanded in a sound decoding thread 35(corresponding to the sound expanding unit 20). In the case of pictureinformation, the information sent out to the ring buffer 33 is expandedin a decoding process 36.

The monitor table is sent out to the ring buffer 34, and is utilized ina monitor thread 37 for determining the sequence for expanding thepicture (corresponding to the control terminal unit 26, pictureexpanding control unit 15, and sound expanding control unit 16 above).The same monitor table is utilized in a picture synthesizing thread 39for synthesizing a picture. The monitor table utilized in the monitorthread 37 reads out the next table from the ring buffer 34 whenexpanding of all sound and pictures is over. The picture informationexpanded in the decoding process 36 (corresponding to the pictureexpanding unit 18) is sent out to a picture single buffer 38. When thetransmitted pictures are all ready, the picture is synthesized by usingthe ratio of the image synthesis controlled in the monitor table, by apicture synthesis thread 39 (corresponding to the picture synthesizingunit 19). The result of synthesis is accumulated in a synthesis buffer41 (corresponding to the synthesis result accumulation unit 22), therebywaiting in display wait state until reaching the display time in adisplay monitor thread 42 (corresponding to the reproduction timecontrol unit 23).

FIG. 5 is a diagram for explaining the composition of information usedin the constitution in FIG. 4. In the example in FIG. 5, the informationreceived from the disk or network has a fixed length of 188 bytes (B).The composition of the sound information separated in the DEMUX thread31 is composed of packet synchronous code, reproduction time, framelength showing the length of sound to be reproduced, and sound data (C).The composition of picture information is composed of packet synchronouscode, frame number for identifying the picture, frame length for showingthe size of picture information, and picture data (D). The inventiondoes not require processing in the unit of one frame, but it is allowedto process in a small block unit such as macro block unit.

The composition of monitor table is composed of picture display time,number of pictures to be displayed (synthesized) in one frame, ID ofeach picture, frame number, priority for expanding or displaying,identifier showing the frame type (I picture, P picture, B picture),display horizontal position, display vertical position, and eachinformation of layer showing the ratio of synthesis (E). Incidentally,the synthesis ratio of picture and synthesis ratio of sound may bechanged by correspondence. For example, when two pictures correspond totwo sounds respectively, if the picture synthesis ratio is α:1−α, thecorresponding sound synthesis ratio may be also α:1−α. Not limited tothe relation between pictures, the relation between sounds may be alsodescribed (for example, direction, kind such as BGM or speech sound).

FIG. 6 is a diagram for explaining the operation of DEMUX thread. From afile or network, data of fixed length of 188 bytes is read in (5-1). Theread data is analyzed, and set in the type of the structure of sound,picture or monitor table as mentioned above (5-2). If possible to writeinto the ring buffer, the sound, picture and monitor tables are writteninto individual ring buffers. The corresponding relation of pictureobject ID and plural picture expanding means is obtained. In thisexample, from the object ID of the earlier number, the data issequentially written out into the shared memory of the earlier ringbuffer number (5-3). The write pointer of the written buffer is advanced(5-4). After writing the picture and sound information for one monitortable, the counter of the semaphore for monitor thread control isupdated (5-5). Thus, the monitor thread is controlled from the DEMUX.

FIG. 7 is a diagram for explaining the operation of the monitor thread.Reading in the monitor table, the read pointer is advanced (6-1).Checking the priority of object in overload, the picture frame of highpriority is investigated (6-2). The content of the monitor table istransferred to the synthesis side thread (6-3). The next step is to waitfor creation of data for the portion of one monitor table from the DEMUX(6-4). In the sequence of higher priority of processing, the framenumber of picture to be displayed is written into the decoding process(6-5), and the present time and display time are compared, and if not intime, without skipping the I frame, only the frame of PB is skipped(6-6). Permitting the execution of the corresponding decoding process(6-7), completion of processing is awaited (6-8).

FIG. 8 is a diagram for explaining the operation of the decodingprocess. The first step is to wait for permission of execution from themonitor thread (7-1). The state of input picture is checked, and it isinvestigated if the picture serial number and the entered frame are tobe skipped or not (7-2). The next step is to wait until the picture datato be decoded is collected in the ring buffer (7-3). If there is nocorresponding picture data in the serial number of the picture indicatedfrom the monitor thread, decoding is skipped, and the read pointer isadvanced (7-4). If the input picture is not skipped, the decoding isprocessed, and the read pointer is advanced (7-5). The decoding resultis issued (7-6), and end of processing is noticed to the monitor thread(7-7).

When expanding different types of picture objects by using the sameprocess (or the same thread, or processor in the case of hardware), bycontrolling in correspondence between the frame number of the pictureexpanded in the past and the image before being expanded in the decodingprocess, it is not necessary to generate and utilize multiple processessimultaneously (at least, only the information relating to theimmediately preceding frame is enough; if there are different types offrame pictures such as I, P, B, the control sequence and output sequenceare different, and therefore such control in the decoding process isneeded).

FIG. 9 is a diagram for explaining the operation of picture synthesisthread. The first step is to wait for monitor table from the monitorthread (8-1). The priority of the picture to be processed is checked(8-2). Next is to wait for the picture of decoding result from thehigher priority (8-3). The picture is synthesized according to thedisplay position (8-4). The synthesizing result is written into thesynthesis buffer (8-5). The picture information to be displayed isselected either by the picture expanding means or the picturesynthesizing means. When skipping the picture object ID that is not tobe displayed, it is necessary to notice that the expanding result is notissued to the picture synthesizing means. Concerning the sound, thesound information to be reproduced can be selected either by the soundexpanding means or the sound synthesizing means.

FIG. 10 is a diagram for explaining the operation of display monitorthread. First is to wait until the synthesis picture is written (9-1).In the case of a first display, the time of starting display is acquired(9-2), and the corresponding relation with the time to be displayed iscontrolled. If not reaching the display time, waiting until reaching thetime, the display of synthesis picture is delayed (9-3).

Referring now to FIG. 11, the user interface of the picture synthesizingapparatus of the invention is described below.

In the example in FIG. 11, the foreground picture is combined with thebackground picture, and the building located at a remote position issynthesized as a translucent image at the synthesizing ratio of 0.5. Asshown in FIG. 11, the picture to be used is not required to betwo-dimensional picture. The helicopter and balloon shown in theforeground as three-dimensional image are synthesized with thebackground in two-dimensional image. The helicopter and balloon in theforeground are not always required to be three-dimensional images. Thepicture may be expressed as two-dimensional image when located at aremote position (which may be defined in the size displayed in twodimensions on the screen; for example, if smaller than the size of 20dots×20 dots, the object may be defined to be located at a remoteposition), and as three-dimensional image when located at a nearposition. The picture to be mapped in the wire-frame model ofthree-dimensional image is not limited to still picture, but may be alsoa motion picture. Concerning the picture quality, by setting the picturequality high in the central portion and lower in peripheral parts, thenecessary information demanded by the user can be selectivelytransmitted by priority (by changing the picture quality depending onthe picture synthesizing position in this way, it is expected to enhancethe response). Besides, in the case of three-dimensional picture, thepriority may be set low in the picture displayed remotely and thepriority may be set high in the image displayed closely. The control ofpicture quality can be realized by changing the quantization step.

FIG. 12 is a diagram explaining the method of picture transmissiondepending on fluctuation of capacity of the reception side terminal.Herein, to prevent overload of processing of reception terminal as thepictures to be transmitted are increased, the method of controlling andmanaging including the compression apparatus is described. For example,in the video-on-demand system of MPEG2 base realized by the hardware,the transmission side terminal confirms the performance of the receptionside terminal (for example, the image compression method, size andcommunication protocol) mutually before transmitting and receiving videoinformation. Accordingly, at the transmission side terminal, since theprocessing capacity of the reception side terminal is nearlyestablished, it is not necessary to monitor sequentially the receptionstate or reproduction state of the reception side terminal.

On the other hand, when realizing picture compression and expansion bythe hardware, the number of pictures to be compressed and expanded atthe terminal is fixed. However, when realizing the picture compressionand expansion by the software, the number of pictures to be compressedand expanded at the terminal can be dynamically varied. When compressingor expanding the picture in the multitask environments by the software,effects are large by the picture size, the quantization parameter forcompressing the picture, and the object picture (whether intra-framecoding or inter-frame coding, content of the taken picture), and others,and the picture size to be processed (compressed, expanded) at theterminal and the number of pictures to be processed simultaneously varywith the time. Accordingly, at the transmission side terminal, unlesssequentially determining the picture compression method (method ofpicture compression, presence or absence of picture compression,quantization step, priority of compression, picture size to becompressed, etc.) depending on the reception situation of reception sideterminal (for example, capacity of reception buffer, priority ofreproduction of picture, response time of reception confirmation), anddetermination of priority when the reception terminal is overloaded, itmay exceed the capacity of the reception side, finally leading tobreakdown.

For example, as shown in FIG. 12( b), if the capacity of the receptionbuffer of the reception side terminal exceeds 80%, the nearly overflowstate of the reception buffer is noticed to the transmission side, andthe measure is executed by properly selecting and combining the methodof picture compression (for example, changing from MPEG1 to run length,transmission quantity of compressed picture is decreased), the method oflimiting the transmission quantity by presence or absence of compressedpicture (compressing the picture, and stopping transmissiontemporarily), change of priority of compression (if there are pluralprocesses to be compressed, the priority for compressing is lowered, andthe transmission quantity of picture to be compressed is decreased),change of picture size (changing the compression size from CIF tosmaller QCIF, the transmission quantity of compressed picture isdecreased), or change of quantization step (decreasing the transmissionquantity of compressed picture by changing the picture quality), themethod of adjusting the number of frames (decreasing the number offrames to be processed), and the method of determining the priority whenthe reception terminal is overloaded. As a result, overflow of thereception buffer at the reception side terminal is avoided.

Similarly, when the capacity of the reception buffer at the receptionside is under 20%, the nearly underflow state of the reception buffer atthe reception side terminal is noticed to the transmission side, and inthe reverse manner of the above, at the transmission side terminal, themethod of picture compression, presence or absence of picturecompression, priority of picture compression, picture size, quantizationstep, and number of frames are properly selected and combined, andexecuted. By thus executing the method of increasing the transmissionquantity, the underflow of the reception buffer at the reception sideterminal can be avoided.

Aside from monitoring of the state of the reception buffer, if thereproduction capacity at the reception side terminal is limited andthere are plural pictures to be reproduced, it is necessary to determinedefinitely the pictures to be reproduced by priority by the user at thereception side terminal, or determine automatically all pictures to bereproduced by priority at the terminal side (the pictures to bereproduced by priority must be registered by the user preliminarily inthe reception terminal as the rule; for example, the priority may begiven to the picture of smaller size, or the reproduction interval islow in the pictures displayed in the background). It may be easilyrealized by noticing the load of the reception side terminal (forexample, the occupation time of CPU necessary for reproduction) to thetransmission side terminal.

When the reproduction load of the reception side terminal exceeds 80% ofthe processing capacity of the terminal, it is noticed to thetransmission side that the reception side terminal is overloaded, andreceiving this notice, at the transmission side, in order to lower thelower to be processed at the reception side terminal in the same manneras above, the processing capacity at the reception side terminal isalleviated by properly selecting, combining and executing the method ofpicture compression (for example, changing from MPEG1 to run length,processing amount is decreased), the method by presence or absence ofcompressed picture (compressing the picture, and stopping transmissiontemporarily), change of priority of compression (the priority forcompressing is lowered for the picture lower in importance, and pictureof higher importance is compressed and sent out by priority), change ofpicture size (changing the compression size from CIF to smaller QCIF,the reproduction side load is decreased), or change of quantization step(decreasing the transmission quantity of compressed picture by changingthe picture quality), the method of adjusting the number of frames, andthe method of processing according to the priority in overloadedprocessing.

To the contrary, when the load is under 20% of the processing capacityof the reception side terminal, it is known there is an allowance in theprocessing capacity at the reception side terminal, and in the reversemanner of the above, at the transmission side terminal, the method ofpicture compression, presence or absence of picture compression,priority of picture compression, picture size, quantization step, andnumber of frames are properly selected and combined, and executed, sothat the picture of high quality and short frame intervals is sent outto the reception side terminal. As a result, picture transmission makingthe use of the capacity of the reception side terminal is realized.

Finally, the processing situation of the reception side terminal isknown from the response time of confirmation of reception from thepicture synthesizing apparatus at the reception side. For example, whenvideo data is sent out from the transmission side terminal to thereception side terminal, the reception side terminal replies thereception of video data or completion of decoding process, synthesizingor display process to the transmission side terminal, and supposing thisresponse time to be, for example, within 1 second as an ordinary value,if the load increases at the reception side terminal, the response timeis extended to, say, 5 seconds (the ordinary value may be once measuredwhen connecting the terminals, or may be measured periodically duringcommunication, or it may be instructed by the user; the response timemay be measured periodically, or the measuring interval may be varied inrelation to the terminal load or the previous result of response time).By the change of this response time, the above method of picturecompression, presence or absence of picture compression, priority ofpicture compression, picture size, and quantization step may be properlyselected, combined and executed, so that the load at the receptionterminal may be decreased, thereby shortening the response time (seecase 1 in FIG. 16). The same processing may be done by receiving thereproduction time or decoding time at the reception terminal.

As the method in consideration of the terminal state of the receptionside, the methods of measuring the capacity of reception buffer at thereception side terminal, load of reception side terminal, and responsetime of reception side terminal mentioned above may not be employedalone, but may be properly selected and combined (the same method can beapplied to the sound). Moreover, by transmitting the information aboutthe picture and sound processed on the basis of the priority informationat the reception side terminal (in the presence of plural picturestreams or sound streams, the information showing which stream is thepicture or sound stream actually processed at the reception sideterminal, or how many seconds is the duration of the reproduced imagestream) to the transmission destination through communication route, thevideo data transmission from the transmission side to the reception sideterminal is prevented from exceeding the processing capacity at thereception terminal (see case 2 in FIG. 16; by knowing the actuallyprocessed video data, the information quantity such as quantizationparameter and picture size at the transmission side can be known; inthis example, the processing is fed back in the unit of frame, but inthe case of H. 263, as mentioned above, it may be the picture unithandled independently such as GOB). The same method is applied to thesound.

FIG. 13 is a diagram for explaining a picture compression apparatus inan embodiment of the invention. This embodiment refers to picture, butit is similarly applied to compression of sound. In the example in FIG.13, by changing the quantization step in every picture input means 1207,or by following up and changing the quantization step when the receptionsituation at the reception side terminal is changed by the control onthe picture input means 1207, it is intended to decrease the incrementof the generation quantity of compressed picture. The picturecompression apparatus in FIG. 13 comprises a quantization step controlunit 1201 for controlling the information about the quantization step, apicture input control unit 1202 for controlling the control state of thepicture input means 1207, other terminal control request control unit1203 for monitoring the situation of the reception buffer of thereception side terminal device, an operation control unit 1204 forrecording and controlling the time transition of the control, a picturecompression unit 1205 as means for compressing the picture, an outputunit 1206 for delivering the result of compression to the communicationroute or memory device, picture input means 1207 for feeding thepicture, and picture processing decision control means 1208 forcontrolling these parts and controlling the management.

The method of picture compression may include both standardized methodssuch as JPEG, MPEG1/2, H. 261, H. 263, and non-standardized methods suchas wavelet and fractal. The picture input means 1207 may be camera, orrecording device such as video or optical disk.

As the method of using this picture compression apparatus, in the casethe picture input means 1207 is a camera, when the camera at thetransmission side terminal is operated by the reception side terminal,or the camera is operated at the transmission side, the picture qualitychanges substantially, and the transmission coding quantity fluctuates.For example, when the camera contrast is raised, the picture is easierto see, but the transmission coding quantity increases. Accordingly, todecrease the coding quantity while the contrast is enhanced as mentionedabove, the coding quantity can be suppressed by properly selecting,combining and executing the method of picture compression, presence orabsence of picture compression, priority of picture compression, picturesize, quantization step, and the number of steps.

Herein, the camera operation includes the came moving direction (pan,tilt, zoom), contrast, focus, and camera position (for example, thecamera is directed downward when taking a drawing, and horizontal whentaking a human figure). As the method of changing the method of picturecompression, when the camera is directed downward, it is judged thatdocument image is being taken, and the picture is transmitted in runlength, and when the camera is in horizontal direction, it is judgedthat the human face is taken, the picture is taken and transmitted by H.261. As a result, transmission of unnecessary information can bereduced.

When plural cameras are used and pictures obtained from plural camerasare transmitted, if the communication capacity is limited, the picturequality and number of frames of the image from the camera being noticedby the user may be increased, while the picture quality and number offrames of the image from other cameras may be decreased. By operatingthe picture quality and number of frames of the image obtained from thecamera being noticed, the quantity of information increases, and it isnecessary to adjust the quantity of generated information by limitingthe picture obtained from other cameras accordingly. As the method ofadjusting the quantity of generated information, the picture size, thequantization step, or the number of frames may be adjusted. An exampleof creating a panoramic picture by using plural cameras is describedlater in FIG. 15.

FIG. 14 is an example of information controlled by the operation controlunit 1204. In the example in FIG. 14, the picture size, camera control,control request of other terminal, quantization step, and number offrames, not shown, are controlled. On the basis of such controlinformation, the relation between the quantization step and cameraoperation is recorded and control as history information so that thereception buffer at the reception side terminal may not overflow, andtherefore the limit about camera operation can be given to the user. Byautomatically changing the quantization step, picture size, and numberof frames, overflow or underflow of reception buffer at the receptionside terminal due to camera operation can be prevented.

FIG. 15 shows an example of application of the same picture compressionapparatus in creation of panoramic picture. In the example in FIG. 15,the pictures entered from plural cameras are acquired in the input unit1407. When the plural obtained pictures are joined (synthesized) withoutseam at the reception terminal 1408 side, if the reception terminal 1408is overloaded, the terminal is broken down, and to prevent this, thepictures are provided with the priority defining the sequence ofpictures to be processed in the case of overload of the receptionterminal. As a result, the reception terminal 1408 side is preventedfrom being overloaded.

The picture compression apparatus shown in FIG. 15 comprises an inputunit 1407 having plural (N) cameras, a priority decision control unit1401 for providing the pictures obtained in the input unit 1407 with thepriority, an operation history control unit 1402 for controlling theoperation history of the cameras instructed and manipulated by the user(in order to notice in particular), a picture quality control unit 1403for controlling the picture quality of the image, a picture synthesisunit 1404 for synthesizing the pictures obtained from the camerasaccording to the priority (not synthesizing pictures of low priority),an output unit 1405 for delivering the result of synthesis, and acompression control unit 1406 for controlling these parts. The outputunit 1405 is connected to the reception terminal 1408 throughcommunication route.

The output destination of the output unit 1405 may be either recordingdevice or communication route. Synthesis of pictures may not be alwaysdone at the transmission side terminal. The pictures provided with thepriority may be sent to the reception side terminal through thecommunication route, and synthesized at the reception terminal side.When the plural obtained pictures are synthesized at the transmissionside terminal and reproduced at the reception side terminal, theobtained pictures are synthesized at the transmission side in thesequence of priority necessary (for display) at the reception terminal,and the synthesized picture is transmitted to the reception terminaldevice through the transmission route.

As the method of assigning the priority, the higher priority or higherpicture quality (for example, greater number of frames or higherresolution) may be assigned in the sequence of the picture obtained bythe camera instructed by the user, or the picture obtained from thecamera most often instructed in the past (the picture of high prioritymay not be always set at high picture quality). As a result, the pictureof the higher degree of notice by the user is displayed at high picturequality by priority. Depending on the priority assigned to the picture,the picture transmission from the transmission side terminal may becontrolled, or by controlling the expanding or displaying of the pictureat the reception side terminal, the response of the terminal at the usercan be assured.

Besides, sequentially from the picture higher in priority, higher inpicture quality, or greater in the number of frames, the priority andpicture quality are lowered in steps with respect to the adjacent joinedpicture (the priority may be controlled at the transmission sideterminal, or reception side terminal). The method of determining thepriority may not be always based on the operation history of cameras. Asmentioned above, the priority may be determined on the basis of thelocal decoding time required for compression, or the execution rate forspecifying the number of times of execution of processing may be definedin the peripheral pictures, sequentially from the picture higher inpriority, higher in picture quality, or greater in the number of frames.As for the sound, by providing the plural cameras with microphonesindividually, and controlling the presence or absence of compression ofsound, only the sound corresponding to the picture in the directionnoticed by the user can be synthesized.

Further, as mentioned above, by referring to the response time betweenthe transmission side terminal and reception side terminal, thequantization step or number of frames may be determined. By transmittingthe information about the picture processed on the basis of the priorityinformation when the reception side terminal is overloaded to thetransmission destination through the communication route, the picturedata transmission from the transmission side to the reception sideterminal is prevented from exceeding the processing capacity of thereception terminal. Also by transmitting the frame skip state at thereception terminal to the transmission side, the data quantity can beadjusted depending on the state.

Moreover, by transmitting the picture by the transmission methodaccompanied by retransmission, and transmitting the sound by thetransmission method not accompanied by retransmission, it is constitutedso that the reception side terminal transmits any information out of theretransmission count of picture, error rate of received sound, andinformation about discarding rate, to the transmission side terminal.Thus, as the transmission side terminal, by determining any one of thepicture compression method, the quantization step, number of frames,size of picture to be compressed, and presence or absence of picturecompression, it is possible to control to minimize delay of soundtransmission without disturbing the picture. For example, in thecommunication using the TCP/IP, the picture transmission is realized bythe TCP, and the sound transmission by UDP (the picture and sound may betransmitted physically in the same transmission route or not). Thecommunication method is not limited to the TCP/IP alone. In this method,when transmitting plural pictures or sounds simultaneously, thediscarding rate or error rate may be defined in each sound, and pluralpicture compression methods or transmission methods may be controlled.

Lastly, in the case of picture transmission at low bit rate by usinganalog telephone circuit or large fluctuation of picture data, usually,a large block noise occurs in the picture. In such a case, it is hard tomaintain the picture quality by the compression process alone.Accordingly, by using a filter which passes only low frequency signal(for example, low pass filter by picture processing, or physicalpolarizing filter) in the monitor at the picture output side, althoughthe picture seems less sharp, the picture free from noise orobjectionable pattern is obtained.

INDUSTRIAL APPLICABILITY

As clear from the description herein, when decoding or synthesizingplural pictures or sounds simultaneously, it is advantageous that theprocessing quantity can be controlled on the basis of the prioritydepending on the load situation at the terminal.

The invention is also beneficial that plural pictures or sounds can besynthesized depending on the accounting situation.

1. A picture decoding and coding apparatus comprising: a picture codingapparatus including a picture coding unit operable to code pictures andprovide a picture identifier for each picture as an I, P or B picture, apriority providing unit operable to correlate each coded picture with apriority identifier which is independent of the picture identifier, anda transmission control unit operable to transmit or store the codedpictures with the priority identifiers, and a picture decoding apparatusincluding a reception control unit operable to receive or read the codedpictures, and a picture decoding unit operable to decode the codedpictures with the priority identifiers, wherein each priority identifieris used by the picture decoding apparatus to determine whether eachpicture should be processed or not be processed according to aprocessing capacity of the picture decoding apparatus, and each priorityidentifier is used independently of the picture identifiers andindependently of whether the picture is an I, P or B picture.
 2. Apicture decoding and coding apparatus according to claim 1, wherein thepriority identifiers provided for the I pictures include at least twodifferent levels of priority.
 3. A picture decoding and coding apparatusaccording to claim 1, wherein the priority identifiers provided forinter-frame coded pictures and intra-frame coded pictures include afirst priority level and a second priority level, respectively, whichare different from each other.
 4. A picture coding apparatus comprising:a picture coding unit operable to code pictures and provide a pictureidentifier for each picture as an I, P or B picture, a priorityproviding unit operable to correlate each coded picture with a priorityidentifier which is independent of the picture identifier; and atransmission control unit operable to transmit or store the codedpictures with the priority identifiers to a picture decoding apparatus,wherein each priority identifier is used by the picture decodingapparatus to determine whether each picture should be processed or notbe processed according to a processing capacity of the picture decodingapparatus and each priority identifier is used independently of thepicture identifiers and independently of whether the picture is an I, Por B picture.
 5. A picture coding apparatus according to claim 4,wherein the priority identifiers provided for the I pictures include atleast two different levels of priority.
 6. A picture coding apparatusaccording to claim 4, wherein the priority identifiers provided forinter-frame coded pictures and intra-frame coded pictures include afirst priority level and a second priority level, respectively, whichare different from each other.
 7. A picture decoding apparatuscomprising: a reception control unit operable to receive or read pictureidentifiers identifying an I, P or B picture for coded pictures withpriority identifiers transmitted from a picture coding apparatus, andeach picture; and a picture decoding unit operable to decode the codedpictures with the priority identifiers, wherein each priority identifieris independent of the picture identifiers and is used by the picturedecoding apparatus to determine whether each picture should be processedor not be processed according to a processing capacity of the picturedecoding apparatus and each priority identifier is used independently ofthe picture identifiers and independently of whether the picture is anI, P or B picture.
 8. A picture decoding apparatus according to claim 7,wherein the priority identifiers provided for the I pictures include atleast two different levels of priority.
 9. A picture decoding and codingapparatus according to claim 7, wherein the priority identifiersprovided for inter-frame coded pictures and intra-frame coded picturesinclude a first priority level and a second priority level,respectively, which are different from each other.
 10. A picturedecoding and coding apparatus comprising: a picture coding apparatusincluding; a picture coding unit operable to code pictures and provide apicture identifier for each picture as an I, P or B picture, a priorityproviding unit operable to correlate each coded picture with a priorityidentifier which is independent of the picture identifier, and atransmission control unit operable to transmit or store the codedpictures with the priority identifiers, and a picture decoding apparatusincluding; a reception control unit operable to receive or read thecoded pictures, and a picture decoding unit operable to receive thecoded pictures with the priority identifiers, wherein each priorityidentifier is used by the picture decoding apparatus to determinewhether each picture should be processed or not be processed accordingto a processing load of the picture decoding apparatus and each priorityidentifier is used independently of the picture identifiers andindependently of whether the picture is a I, P or B picture.
 11. Apicture decoding and coding apparatus according to claim 10, wherein thepriority identifiers provided for the I pictures include at least twodifferent levels of priority.
 12. A picture decoding and codingapparatus according to claim 10, wherein the priority identifiersprovided for inter-frame coded pictures and intra-frame coded picturesinclude a first priority level and a second priority level,respectively, which are different from each other.
 13. A picture codingapparatus comprising: a picture coding unit operable to code picturesand provide a picture identifier for each picture as an I, P or Bpictures; a priority providing unit operable to correlate each codedpicture with a priority identifier which is independent of the pictureidentifier; and a transmission control unit operable to transmit orstore the coded pictures with the priority identifiers to a picturedecoding apparatus, wherein each priority identifier is used by thepicture decoding apparatus to determine whether each picture should beprocessed or not be processed according to a processing load of thepicture decoding apparatus, and each priority identifier is usedindependently of the picture identifiers and independently of whetherthe picture is an I, P or B picture.
 14. A picture coding apparatusaccording to claim 13, wherein the priority identifiers provided for theI pictures include at least two different levels of priority.
 15. Apicture coding apparatus according to claim 13, wherein the priorityidentifiers provided for inter-frame coded pictures and intra-framecoded pictures include a first priority level and a second prioritylevel, respectively, which are different from each other.
 16. A picturedecoding apparatus comprising: a reception control unit operable toreceive or read picture identifiers identifying an I, P or B picture forcoded pictures with priority identifiers transmitted from a picturecoding apparatus, and each picture; and a picture decoding unit operableto decode the coded pictures with the priority identifiers, wherein eachpriority identifier is independent of the picture identifiers and isused by the picture decoding apparatus to determine whether each pictureshould be processed or not be processed according to a processing loadof the picture decoding apparatus, and each priority identifier is usedindependently of the picture identifiers and independently of whetherthe picture is an I, P or B picture.
 17. A picture decoding apparatusaccording to claim 16, wherein the priority identifiers provided for theI pictures include at least two different levels of priority.
 18. Apicture decoding apparatus according to claim 16, wherein the priorityidentifiers provided for inter-frame coded pictures and intra-framecoded pictures include a first priority level and a second prioritylevel, respectively, which are different from each other.
 19. A picturedecoding and coding apparatus comprising: a picture coding apparatusincluding; a picture coding unit operable to code pictures and provide apicture identifier for each picture as an I, P or B picture, a priorityproviding unit operable to correlate each coded picture with a priorityidentifier which is independent of the picture identifier, and atransmission control unit operable to transmit or store the codedpictures with the priority identifiers, and a picture decoding apparatusincluding; a reception control unit operable to receive or read thecoded pictures, and a picture decoding unit operable to decode the codedpictures with the priority identifiers, wherein each priority identifieris used by the picture decoding apparatus to determine whether eachpicture should be processed or not be processed when the picturedecoding apparatus is overloaded, and each priority identifier is usedindependently of the picture identifiers and independently of whetherthe picture is an I, P or B picture.
 20. A picture decoding and codingapparatus according to claim 19, wherein the priority identifiersprovided for the I pictures include at least two different levels ofpriority.
 21. A picture decoding and coding apparatus according to claim19, wherein the priority identifiers provided for inter-frame codedpictures and intra-frame coded pictures include a first priority leveland a second priority level, respectively, which are different from eachother.
 22. A picture coding apparatus comprising: a picture coding unitoperable to code pictures and provide a picture identifier for eachpicture as an I, P or B picture; a priority providing unit operable tocorrelate each coded picture with a priority identifier which isindependent of the picture identifier; and a transmission control unitoperable to transmit or store the coded pictures with the priorityidentifiers to a picture decoding apparatus, wherein each priorityidentifier is used by the picture decoding apparatus to determinewhether each picture should be processed or not be processed when thepicture decoding apparatus is overloaded, and each priority identifieris used independently of the picture identifiers and independently ofwhether the picture is an I, P or B picture.
 23. A picture codingapparatus according to claim 22, wherein the priority identifiersprovided for the I pictures include at least two different levels ofpriority.
 24. A picture coding apparatus according to claim 22, whereinthe priority identifiers provided for inter-frame coded pictures andintra-frame coded pictures include a first priority level and a secondpriority level, respectively, which are different from each other.
 25. Apicture decoding apparatus comprising: a reception control unit operableto receive or read picture identifiers identifying an I, P or B picturefor coded pictures with priority identifiers transmitted from a picturecoding apparatus, and each picture; and a picture decoding unit operableto decode the coded pictures with the priority identifiers, wherein eachpriority identifier is independent of the picture identifiers and isused by the picture decoding apparatus to determine whether each pictureshould be processed or not be processed when the picture decodingapparatus is overloaded, and each priority identifier is usedindependently of the picture identifiers and independently of whetherthe picture is an I, P or B picture.
 26. A picture decoding apparatusaccording to claim 25, wherein the priority identifiers provided for theI pictures include at least two different levels of priority.
 27. Apicture decoding apparatus according to claim 25, wherein the priorityidentifiers provided for inter-frame coded pictures and intra-framecoded pictures include a first priority level and a second prioritylevel, respectively, which are different from each other.